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(54) Measuring simultaneous multiple judgments by multiple subjects 

(57) A microcomputer is connected to several input devices, for example, computer joysticks or dials, via 
interface cards and to a video tape recorder and video monitor. Computer software allows automatic and 
real-time collection from the input devices of multiple subjects immediate or current continuous like/dislike 
and/or other judgments of a sequence of stimuli or continuous actions presented on the video monitor. The 
judgements are collected into computer files in a form which may be graphically presented or statistically 
analysed without further processing. 

Optionally, via interface cards, the software also permits control of the video recorder by computer 
keyboard or input devices, presentation of overlays on the monitor, and control of these overlays by computer 
keyboard or input devices. The input devices may transmit data as encoded signals to the computer via 
telecommunications. 
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AN APPARATUS TO MEASURE JUDGEMENTS 



1. OVERVIEW 



The invention comprises a novel conjunction of an input device, a microcomputer and a video 
tape recorder controlled through appropriate software and interface cards. These are used to 
collect automatically several subjects' immediate or current like/dislike or other judgments of 
a sequence of stimuli or continuous actions presented on a video screen. The judgments are 
collected by the computer in a form which may be easily statistically analysed without further 
processing. 

In the preferred embodiment the apparatus consists of a microcomputer, a series of interface 
cards, a video recorder, and several games or other "joysticks" as input devices. These items 
are connected and controlled through the use of computer software which permit: 

1 Direct data collection from the multiple input devices into computer files. 

2 Data collection of several measures simultaneously from each subject. 

3 Control of the video recorder mechanism by the computer keyboard or input device, e.g., 

to pause, review or to go fast forward or backward. 

4 Control of the computer data collection by signals from the video recorder, i.e., to start and 

stop data collection. 

5 Time coding of each frame so that any frame may be located by instructions given through 

the keyboard or input device. 

6 Definition of parameters through the keyboard, e.g., subjects' names or which scales to be 

used before running the video. 

In other embodiments: 

1 Only some of these functions may be implemented. For example, 1.1 the input device may 

not be able to control the video recorder mechanism, or 1.2, the video recorder may be 
omitted and replaced by a television broadcasting the material to be judged by subjects 
using the input devices. 

2 Sliders, touch pads, keyboards, computer "mouse" are examples of other input devices that 

may be used. 

3 Presentation of overlays on the video screen, e.g., pointers, graphical, numerical or textual 

displays. 

4 Control of these overlays by the computer keyboard or input device, e.g., to move a 

graphical display or to remove or display a scale or questionnaire. 

5 Subjects may be operate the input devices remote from the computer, the signals from the 

input devices being encoded (using "modems") for transmission via telecommunications, 
e.g. , telephone lines. 
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2. BACKGROUND TO THE INVENTION 



The methods of measuring human judgements, e.g., affective or intensity responses to stimuli 
rely predominantly on the use of the self-report questionnaire or interview. The scales used 



These methods are unsuitable for collecting responses to continuous tasks or activities 
involving stimuli. If subjects judge a sequence of events as they occur, their attention is 
necessarily divided between observing the stimuli and completing the questionnaire. If 
subjects rate a sequence of events afterwards, there may be errors in recall due to recency 
effects. In interviews, subjects' judgements may be influenced unpredictably by reactions to 
social aspects of the situation, e.g., wishing to impress or please the interviewer. 

Zajonc (1980) points out that affective responses cannot be adequately verbalised. He 
suggests that affect is a primitive response fundamentally related to approach/avoidance 
behaviour and is therefore best measured by direct motor responses rather than by verbally 
and socially mediated procedures such as self-report (Zajonc and Markus, 1982). 

In clinical studies mood has been measured across time by subjects drawing a line from left 
to right down a page whilst moving a pen or pencil downwards. This suffers from 
synchronicity errors since the hand may not move down the page at an even speed. The data 
is also hard to quantify. 

In Newtson's studies (1971) subjects used a push button to decide on the points where actions 
displayed on a video screen could be "meaningfully" separated. In these studies, 
assessments were being made as the action occurred, however, the data was of a nominal 
yes/no character, and data analysis was made by manually going through the videotapes and 
noting the time of the button presses, a laborious procedure. In addition, tape stretch and 
errors in timing may introduce synchronicity errors. 

These methods to measure judgements across sequences suffer from errors due to lack of 
attention, appropriate measures and synchronicity, lack of detailed information and 
difficulties in summarising and analysing the data. 

Computer controlled methods have been devised to counter these problems. Hughes (1990) 
describes a device in which a subject turns a dial mounted on a hand-held computer to 
register and collect his judgements. This device does not allow synchronicity of 
measurement or rapid analysis of data. Kirk-Smith (1990) describes a device in which a 
computer mouse or other input device attached to a computer is used by a subject to register 
and record judgements, with feedback being provided by on-screen overlays or a visual 
display external to the screen. This device is for single subject use alone, thus slowing data 
acquisition. 

In summary, these computer-based methods do not allow real-time simultaneous recording of 
judgements from several subjects by one computer or the measurement of multiple 
judgements. 
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3. STATEMENT OF INVENTION 

Thus according to the invention there is provided an apparatus for measuring and recording 
multiple judgment responses of several subjects, which comprises: 

(a) Input devices adapted to be activated by the subjects so that their multiple judgment 
responses are converted into measurable and separable signals. 

(b) A connecting device which takes the signals from (a) and passes signals to a device which 
registers the signals and correlates them with the matters currently being judged. 

(c) A recording device which accepts the signals from (1) and records them in a manner 
which permits correlation with the matter being judged by the subject. 

The invention provides a procedure or method comprising the use of the same step or 
components. 

Preferably, the input devices are computer games "joysticks". The connecting device (b), 
may, if desired, be combined with the recording device (c), as, for example, in a computer 
system. 

The recording of subjects' judgment signals may be recorded in a computer, by conventional 
means, in a manner which establishes both the judgment measures and some parameters 
indicative of what was being judged at that moment. 

In this way, the computer will contain a record indicating both the time and the values of the 
judgments. By also feeding into the computer additional data which are indicative of the 
matter being judged, the whole of the desired data for future collation, assessment and 
analysis of the judgments and what they apply to can be assembled. For example, if the 
matter being judged is a video recorded subject, the datum concerning it may be the position 
on the recording tape at which there is a recording to which the judgments relate. 

When the item is not fore-recorded (for example, in a TV transmission or radio transmission, 
or a "live" activity which may be viewed and recorded through a recording device, e.g., a 
video camera) then the matter may be recorded and the relevant data concerning the 
recording of the matter may be treated as additional inputs to the computer. 

To process the results in the desired way, the computer may be operated to produce the 
matter being judged (and, if necessary, also display it) and process the judgment 
measurement data to correlate them with the matter provoking it, for example, in the form of 
a graph or chart. 



4. EMBODIMENTS OF THE INVENTION 



An example of the results of the use of the apparatus in determining peoples' judgements, in 
this case of a video recording of an advertisement, is given in Appendix I. 

An example of the detailed design of a device constructed under the invention is given in 
Appendix II. 



5. APPLICATIONS 



The apparatus has applications which were previously not technically possible. Uses 
involving affective measurement include: 

1 The evaluation and judgment of advertisements. Liked or disliked events or stimuli during 
advertisements may be located, e.g., by measuring subjects' movement of the joysticks in a 
horizontal direction. Other judgments may be measured by this movement, through 
instruction to the subjects. 

2 Other judgements, e.g., of believability, may be collected by simultaneously measuring 
subjects' movement of the joysticks in a vertical direction. Such multiple measures allow 
examination of the correlation of different judgments. 

3 The on-line measurement of fine-grained (e.g., 100 point or greater) judgements from 
remotely situated subjects, e.g., subjects judging broadcast TV at their homes with the 
devices, the signals being carried in modulated form via telephone lines. 

4 Switches and buttons, commonly on computer games joysticks, allow the collection of 
ordinal data pertaining to the viewed sequence, e.g., to identify 'breakpoints' and changes in 
' script 1 . These may also be correlated with the horizontal and/or vertical measures. 

5 The effects of events prior or concurrent to judgments may be investigated as reflected in 
changing judgements, e.g. , the influence of a perfume on judgements. 

6 The diagnosis of phobias. The apparatus allows detailed identification of feared events or 
stimuli in a sequence of events. The effect of interventions on the patient may also be 
evaluated through changes in response. 

7 New product development. The affective aspects of a task may be investigated, e.g., 
through rating the difficulty or liking of a sequence of events. Through this new product 
opportunities, or improvements to old products, may be identified. For example, the use of 
the "pause" control associated with a moving pointer enables subjects to use a joystick to 
identify when and where an attribute is of concern, such as an ergonomic problem associated 
with product use. Similar applications exist for intensity measurements. 
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APPENDIX H: EXEMPLARS OF THE INVENTION 

The apparatus consists of a boxed multi-channel analogue to digital (A/D) converter. Input 
sockets feed in signals from up to 16 standard PC compatible joysticks. Two analogue signals 
(in x and y-axis) and one or more digital (on/off) signals, e.g., from the firing button, are 
sent by each joystick. The apparatus is connected to the PC by a lead to either the parallel 
port or an internal expansion slot. A phono socket feeds in a tone from the audio socket of 
the video recorder to an internal tone detector card within the box. The tone detector is 
connected to a PC compatible via the RS232 port. 

Any tone or sound from the video tape, e.g. , from an advertisement, triggers the tone 
detector, which in turn starts data collection by the computer. The data are collected into an 
computer file and consist data related to the time from the start of the judgement period, and 
the subjects' ratings, at that time. Thus, if the ends of the horizontal travel of the joystick are 
defined as "like" and "dislike", the ends of the vertical travel are defined as "believable" and 
"Unbelievable", and the firing button as "a meaningful change in action", the liking and 
believability at each point over a period of time may be directly collected, as well as 
perceived changes in action. 

In an alternative configuration, each joystick may have a dedicated analogue to digital card, 
so that a series of such cards, one for each joystick required, may be mounted in racks on a 
chassis. The output from each card is led to the PC by a lead to either the parallel port or an 
internal expansion slot. 

Another exemplar comprises a "rack mounted" system of cards including as well as the A/D 
converter, for example, a Video Overlay card, a Video controller card, a Eurocard and a 
Time code card, with appropriate software enabling the computer to control the video tape 
recorder. 

In this exemplar, the VTR may be controlled by the computer, via stored programs, or input 
device such as keyboard or "mouse". Examples of control are pausing and fast 
back/forward of the VTR. Visual elements created by the computer, such as numbers, 
scales, pointers or questionnaires, may also be displayed on the television or monitor screen 
itself. For example, 

- a "mouse" moved pointer may be used to identify specific parts of a product or scene, 
for example, used when the VTR is paused, 

- different attribute terms may be displayed, 

- different types of scaling (e.g, 1-10) may be presented - questionnaires may be overlaid. 

The subjects' choices may be indicated by moving a pointer to the appropriate position on the 
screen. 

Each "frame" of the video may also be "stamped" a time code linked one-to-one with data 
collected by the computer. Thus the VTR tape can be moved to a specific frame at a specific 
time through control by the computer. This exemplar avoids have errors due to slippage, 
tape stretch and the difficulty physically timing from the start of the habit to where a 
certain score is given. 
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CLAIMS 

1 An apparatus comprising the novel conjunction of a computer connected via an interface 
card to several input devices}, with software that allows real-time collection of multiple 
subjects' immediate and continuous responses to a recorded sequence of presented stimuli or 
continuous actions on one or more aspects of judgement and the simultaneous recording of 
these judgements in digital form to a computer data file or files. 

' 2 An apparatus as claimed in Claim 1 wherein software allows recorded subjects' judgements 
be graphically presented without further processing. 

3 An apparatus as claimed in Claim 1 or Claim 2 wherein software allows recorded subjects* 
judgements to be statistically analysed without further processing. 

4 An apparatus as claimed in Claim 1 wherein the input devices are computer joysticks. 

5 An apparatus as claimed in Claim 1 wherein the input devices are dials. 

6 An apparatus as claimed in Claim 1 wherein an interface card connected between the 
computer and the video recorder allows control of the video recorder by the computer. 

7 An apparatus as claimed in Claim 1 wherein an interface card connected between the 
vS U recoider the recorder aUows contro1 of tne computer through signals from the 

8 An apparatus as claimed in Claim 1 wherein a video overlay card allows the presentation 
on the video monitor of visual elements created by the computer. 

9 An apparatus as claimed in Claim 1 and Claim 8 wherein an interface card allows the input 
devices to alter the form and position of these visual elements and the computer to record 
these changes. 

10 An apparatus as claimed in Claim 1 wherein judgements recorded by the input devices 
may be remotely transmitted as encoded signals to the computer via telecommunication. 
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